Systems and methods for presenting augmented reality in a display of a teleoperational system

ABSTRACT

A system comprises a processor and a memory having computer readable instructions stored thereon. The computer readable instructions, when executed by the processor, cause the system to display a surgical environment image. The surgical environment image includes a virtual control element for controlling a component of a surgical system. The virtual control element includes a real-time image of the component of the surgical system in the surgical environment image. The computer readable instructions also cause the system to display an image of a body part of a user used to interact with the virtual control element, receive a gesture of the body part of the user in a predetermined motion via a gesture based input device, and adjust a setting of the component of the surgical system based on the received gesture.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application62/575,759 filed Oct. 23, 2017, which is incorporated by referenceherein in its entirety.

FIELD

The present disclosure is directed to systems and methods for performinga teleoperational medical procedure and more particularly to systems andmethods for presenting augmented reality in a display of ateleoperational system.

BACKGROUND

Minimally invasive medical techniques are intended to reduce the amountof tissue that is damaged during invasive medical procedures, therebyreducing patient recovery time, discomfort, and harmful side effects.Such minimally invasive techniques may be performed through naturalorifices in a patient anatomy or through one or more surgical incisions.Through these natural orifices or incisions, clinicians may insertmedical tools to reach a target tissue location. Minimally invasivemedical tools include instruments such as therapeutic instruments,diagnostic instruments, and surgical instruments. Minimally invasivemedical tools may also include imaging instruments such as endoscopicinstruments. Imaging instruments provide a user with a field of viewwithin the patient anatomy. Some minimally invasive medical tools andimaging instruments may be teleoperated or otherwise computer-assisted.In existing teleoperational medical systems, the surgeon's view of hisor her own extremities may be blocked by a display at a surgical controlconsole, limiting the surgeon's awareness of body position relative tocontrol input devices. Systems and methods are needed to augment theimages on the display to create better body position awareness.

SUMMARY

The embodiments of the invention are summarized by the claims thatfollow below.

In one embodiment, a method comprises displaying a surgical environmentimage. The surgical environment image includes a virtual control elementfor controlling a component of a surgical system. The method alsoincludes displaying an image of a body part of a user used to interactwith the virtual control element. The method also comprises receiving auser input from the user with a gesture based input device while thebody part interacts with the virtual control element. The method alsocomprises adjusting a setting of the component of the surgical systembased on the received user input.

In another embodiment, a method comprises displaying a surgicalenvironment image including a virtual marking element. The method alsoincludes displaying a body part of a user used to interact with thevirtual marking element. The method also includes receiving a user inputfrom the user with a gesture based input device while the body partinteracts with the virtual marking element and generating a patientanatomy mark on a patient anatomy based on the received user input.

In another embodiment, a method comprises displaying an image of aninternal patient anatomy on a display while a patient is located in asurgical environment. The image of the internal patient anatomy isreceived from a first imaging device. The method also includesdisplaying an image of the surgical environment external of the patientanatomy on the display while the patient is located in the surgicalenvironment. The image of the surgical environment external of thepatient anatomy is received from a second imaging device. The displayedimage of the internal patient anatomy is at least partially surroundedby the displayed image of the external patient anatomy.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory innature and are intended to provide an understanding of the presentdisclosure without limiting the scope of the present disclosure. In thatregard, additional aspects, features, and advantages of the presentdisclosure will be apparent to one skilled in the art from the followingdetailed description.

BRIEF DESCRIPTIONS OF THE DRAWINGS

Aspects of the present disclosure are best understood from the followingdetailed description when read with the accompanying figures. It isemphasized that, in accordance with the standard practice in theindustry, various features are not drawn to scale. In fact, thedimensions of the various features may be arbitrarily increased orreduced for clarity of discussion. In addition, the present disclosuremay repeat reference numerals and/or letters in the various examples.This repetition is for the purpose of simplicity and clarity and doesnot in itself dictate a relationship between the various embodimentsand/or configurations discussed.

FIG. 1A is a schematic view of a teleoperational medical system, inaccordance with an embodiment of the present disclosure.

FIG. 1B is a perspective view of a teleoperational manipulator,according to one example of principles described herein.

FIG. 1C is a perspective view of a surgeon's control console for ateleoperational medical system, in accordance with many embodiments.

FIG. 2 illustrates a surgical environment image augmented with an imageof a user's extremity.

FIG. 3 illustrates a method of adjusting a virtual control element witha cursor image of a user's extremity.

FIG. 4A illustrates a cursor image of a user's extremity interactingwith a virtual control element.

FIG. 4B illustrates a cursor image of a user's extremity interactingwith an image of a component in the surgical environment.

FIG. 5 illustrates a cursor image of a user's extremity interacting withan information icon.

FIG. 6 illustrates a method of patient marking using a gesture basedinput device.

FIG. 7 illustrates an image of a surgical environment with a cursorimage of a user's extremity interacting with an image of markingelements.

FIG. 8 illustrates a method of displaying an image of the internalpatient anatomy at least partially surrounded by the displayed image ofthe external patient anatomy.

FIG. 9 illustrates an image of the internal patient anatomy at leastpartially surrounded by the displayed image of the external patientanatomy.

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the principles of thepresent disclosure, reference will now be made to the embodimentsillustrated in the drawings, and specific language will be used todescribe the same. It will nevertheless be understood that no limitationof the scope of the disclosure is intended. In the following detaileddescription of the aspects of the invention, numerous specific detailsare set forth in order to provide a thorough understanding of thedisclosed embodiments. However, it will be obvious to one skilled in theart that the embodiments of this disclosure may be practiced withoutthese specific details. In other instances well known methods,procedures, components, and circuits have not been described in detailso as not to unnecessarily obscure aspects of the embodiments of theinvention.

Any alterations and further modifications to the described devices,instruments, methods, and any further application of the principles ofthe present disclosure are fully contemplated as would normally occur toone skilled in the art to which the disclosure relates. In particular,it is fully contemplated that the features, components, and/or stepsdescribed with respect to one embodiment may be combined with thefeatures, components, and/or steps described with respect to otherembodiments of the present disclosure. In addition, dimensions providedherein are for specific examples and it is contemplated that differentsizes, dimensions, and/or ratios may be utilized to implement theconcepts of the present disclosure. To avoid needless descriptiverepetition, one or more components or actions described in accordancewith one illustrative embodiment can be used or omitted as applicablefrom other illustrative embodiments. For the sake of brevity, thenumerous iterations of these combinations will not be describedseparately. For simplicity, in some instances the same reference numbersare used throughout the drawings to refer to the same or like parts.

The embodiments below will describe various instruments and portions ofinstruments in terms of their state in three-dimensional space. As usedherein, the term “position” refers to the location of an object or aportion of an object in a three-dimensional space (e.g., three degreesof translational freedom along Cartesian X, Y, Z coordinates). As usedherein, the term “orientation” refers to the rotational placement of anobject or a portion of an object (three degrees of rotationalfreedom—e.g., roll, pitch, and yaw). As used herein, the term “pose”refers to the position of an object or a portion of an object in atleast one degree of translational freedom and to the orientation of thatobject or portion of the object in at least one degree of rotationalfreedom (up to six total degrees of freedom).

Referring to FIG. 1A of the drawings, a teleoperational medical systemfor use in, for example, medical procedures including diagnostic,therapeutic, or surgical procedures, is generally indicated by thereference numeral 10. As will be described, the teleoperational medicalsystems of this disclosure are under the teleoperational control of asurgeon. In alternative embodiments, a teleoperational medical systemmay be under the partial control of a computer programmed to perform theprocedure or sub-procedure. In still other alternative embodiments, afully automated medical system, under the full control of a computerprogrammed to perform the procedure or sub-procedure, may be used toperform procedures or sub-procedures. As shown in FIG. 1A, theteleoperational medical system 10 is positioned in a surgicalenvironment 11 and generally includes a teleoperational assembly 12mounted to or near an operating table O on which a patient P ispositioned. The teleoperational assembly 12 may be referred to as apatient side cart. A medical instrument system 14 and an endoscopicimaging system 15 are operably coupled to the teleoperational assembly12. An operator input system 16 allows a surgeon or other type ofclinician S to view images of or representing the surgical site and tocontrol the operation of the medical instrument system 14 and/or theendoscopic imaging system 15.

The operator input system 16 may be located at a surgeon's console,which is usually located in the same room as operating table O. Itshould be understood, however, that the surgeon S can be located in adifferent room or a completely different building from the patient P.Operator input system 16 generally includes one or more controldevice(s) for controlling the medical instrument system 14. The controldevice(s) may include one or more of any number of a variety of inputdevices, such as hand grips, joysticks, trackballs, data gloves,trigger-guns, foot pedals, hand-operated controllers, voice recognitiondevices, touch screens, body motion or presence sensors, and the like.In some embodiments, the control device(s) will be provided with thesame degrees of freedom as the medical instruments of theteleoperational assembly to provide the surgeon with telepresence, theperception that the control device(s) are integral with the instrumentsso that the surgeon has a strong sense of directly controllinginstruments as if present at the surgical site. In other embodiments,the control device(s) may have more or fewer degrees of freedom than theassociated medical instruments and still provide the surgeon withtelepresence. In some embodiments, the control device(s) are manualinput devices which move with six degrees of freedom, and which may alsoinclude an actuatable handle for actuating instruments (for example, forclosing grasping jaw end effectors, applying an electrical potential toan electrode, delivering a medicinal treatment, and the like).

The teleoperational assembly 12 supports and manipulates the medicalinstrument system 14 while the surgeon S views the surgical site throughthe console 16. An image of the surgical site can be obtained by theendoscopic imaging system 15, such as a stereoscopic endoscope, whichcan be manipulated by the teleoperational assembly 12 to orient theendoscope 15. The number of medical instrument systems 14 used at onetime will generally depend on the diagnostic or surgical procedure andthe space constraints within the operating room among other factors. Theteleoperational assembly 12 may include a kinematic structure of one ormore non-servo controlled links (e.g., one or more links that may bemanually positioned and locked in place, generally referred to as aset-up structure) and a teleoperational manipulator. The teleoperationalassembly 12 includes a plurality of motors that drive inputs on themedical instrument system 14. These motors move in response to commandsfrom the control system (e.g., control system 20). The motors includedrive systems which when coupled to the medical instrument system 14 mayadvance the medical instrument into a naturally or surgically createdanatomical orifice. Other motorized drive systems may move the distalend of the medical instrument in multiple degrees of freedom, which mayinclude three degrees of linear motion (e.g., linear motion along the X,Y, Z Cartesian axes) and in three degrees of rotational motion (e.g.,rotation about the X, Y, Z Cartesian axes). Additionally, the motors canbe used to actuate an articulable end effector of the instrument forgrasping tissue in the jaws of a biopsy device or the like. Instruments14 may include end effectors having a single working member such as ascalpel, a blunt blade, an optical fiber, or an electrode. Other endeffectors may include, for example, forceps, graspers, scissors, or clipappliers.

The teleoperational medical system 10 also includes a control system 20.The control system 20 includes at least one memory 24 and at least oneprocessor 22, and typically a plurality of processors, for effectingcontrol between the medical instrument system 14, the operator inputsystem 16, and other auxiliary systems which may include, for example,imaging systems, audio systems, fluid delivery systems, display systems,illumination systems, steering control systems, irrigation systems,and/or suction systems. The control system 20 can be used to process theimages of the surgical environment from the imaging system 15 forsubsequent display to the surgeon S through the surgeon's console 16.The control system 20 also includes programmed instructions (e.g., acomputer-readable medium storing the instructions) to implement some orall of the methods described in accordance with aspects disclosedherein. While control system 20 is shown as a single block in thesimplified schematic of FIG. 1A, the system may include two or more dataprocessing circuits with one portion of the processing optionally beingperformed on or adjacent the teleoperational assembly 12, anotherportion of the processing being performed at the operator input system16, and the like. Any of a wide variety of centralized or distributeddata processing architectures may be employed. Similarly, the programmedinstructions may be implemented as a number of separate programs orsubroutines, or they may be integrated into a number of other aspects ofthe teleoperational systems described herein. In one embodiment, controlsystem 20 supports wireless communication protocols such as Bluetooth,IrDA, HomeRF, IEEE 802.11, DECT, and Wireless Telemetry.

In some embodiments, control system 20 may include one or more servocontrollers that receive force and/or torque feedback from the medicalinstrument system 14. Responsive to the feedback, the servo controllerstransmit signals to the operator input system 16. The servocontroller(s) may also transmit signals instructing teleoperationalassembly 12 to move the medical instrument system(s) 14 and/orendoscopic imaging system 15 which extend into an internal surgical sitewithin the patient body via openings in the body. Any suitableconventional or specialized servo controller may be used. A servocontroller may be separate from, or integrated with, teleoperationalassembly 12. In some embodiments, the servo controller andteleoperational assembly are provided as part of a teleoperational armcart positioned adjacent to the patient's body.

The control system 20 can be coupled with the endoscope imaging system15 and can include a processor to process captured images for subsequentdisplay, such as to a surgeon on the surgeon's console, or on anothersuitable display located locally and/or remotely. For example, where astereoscopic endoscope is used, the control system 20 can process thecaptured images to present the surgeon with coordinated stereo images ofthe surgical site. Such coordination can include alignment between theopposing images and can include adjusting the stereo working distance ofthe stereoscopic endoscope.

A surgical environment monitoring system, including one or moremonitoring devices such as cameras 27 a, 27 b, 27 c, is positioned inthe surgical environment 11. The cameras 27 a-c may be used to captureimages in the surgical environment 11 outside of the patient P anatomy.For example and as will be described further, the cameras 27 a-c may beused to monitor the extremities of the surgeon S during a procedure. Theimages of the surgeon's hands and feet may be presented to the surgeonthrough a display at the console 16 to assist the surgeon duringtransitions that require movement of the extremities to controloperation of the system 10. The cameras 27 a-c may also or alternativelybe used to capture images of the external patient anatomy, theteleoperational assembly, other equipment in the surgical environment,and personnel in the surgical environment. The cameras 27 a-c may bemounted in any of a variety of ways including on discrete pedestals ortripods, from the ceiling, on equipment in the surgical environmentincluding the orienting platform 53, on the shafts of the instruments 14or endoscope 15 external of the patient anatomy, or on equipment worn bythe surgeon S or by other personnel in the surgical environment, such asa head-mounted camera.

A gesture-based interface (GBI) 29 may also be located in the surgicalenvironment 11. The GBI may be a touch-based interface system such as acomputer tablet or may be a three-dimensional tracking and interfacesystem such as a Leap Motion system available from Leap Motion, Inc. ofSan Francisco, Calif. or such as Kinect from Microsoft Corporation ofRedmond, Wash. Additionally or alternatively, the GBI may be a wearabledevice such as a head-mounted device. The GBI 29 may be used to tracktwo or three-dimensional user inputs from the surgeon S or othersurgical personnel.

A patient side interface (PSI) 26 may be located or locatable near thebedside of the patient. The PSI 26 may allow the surgeon S to approachthe patient and still have access to at least some functionality of theconsole 16 or additional inputs not available at the console 16. The PSI26 may include a display for displaying similar or different images fromthose displayed at the console 16. The PSI 26 may include a head-mounteddisplay system, a boom-mounted display system, or a dome-style displaysystem that provides primary and peripheral images or 360° images of thesurgical environment. The PSI 26 may also include a user input devicesuch as a computer tablet, trackball, or three-dimensional input system.In some embodiments, the PSI 26 may include all or some components ofthe GBI 29.

In alternative embodiments, the teleoperational system may include morethan one teleoperational assembly and/or more than one operator inputsystem. The exact number of manipulator assemblies will depend on thesurgical procedure and the space constraints within the operating room,among other factors. The operator input systems may be co-located, orthey may be positioned in separate locations. Multiple operator inputsystems allow more than one operator to control one or more manipulatorassemblies in various combinations.

FIG. 1B is a perspective view of one embodiment of a teleoperationalassembly 12 which may be referred to as a patient side cart. The patientside cart 12 shown provides for the manipulation of three surgical tools30 a, 30 b, 30 c (e.g., instrument systems 14) and an imaging device 28(e.g., endoscopic imaging system 15), such as a stereoscopic endoscopeused for the capture of images of the site of the procedure. The imagingdevice may transmit signals over a cable 56 to the control system 20.Manipulation is provided by teleoperative mechanisms having a number ofjoints. The imaging device 28 and the surgical tools 30 a-c can bepositioned and manipulated through incisions in the patient so that akinematic remote center is maintained at the incision to minimize thesize of the incision. Images of the surgical environment within thepatient anatomy can include images of the distal ends of the surgicaltools 30 a-c when they are positioned within the field-of-view of theimaging device 28.

The patient side cart 12 includes a drivable base 58. The drivable base58 is connected to a telescoping column 57, which allows for adjustmentof the height of the arms 54. The arms 54 may include a rotating joint55 that both rotates and moves up and down. Each of the arms 54 may beconnected to an orienting platform 53. The orienting platform 53 may becapable of 360 degrees of rotation. The patient side cart 12 may alsoinclude a telescoping horizontal cantilever 52 for moving the orientingplatform 53 in a horizontal direction.

In the present example, each of the arms 54 connects to a manipulatorarm 51. Each manipulator arms 51 may connect to a respective one of themedical tools 30 a-c or to the imaging device 28. The manipulator arms51 may be teleoperable. In some examples, the arms 54 connecting to theorienting platform are not teleoperable. Rather, such arms 54 arepositioned as desired before the surgeon 18 begins operation with theteleoperative components.

Endoscopic imaging systems (e.g., systems 15, 28) may be provided in avariety of configurations including rigid or flexible endoscopes. Rigidendoscopes include a rigid tube housing a relay lens system fortransmitting an image from a distal end to a proximal end of theendoscope. Flexible endoscopes transmit images using one or moreflexible optical fibers. Digital image based endoscopes have a “chip onthe tip” design in which a distal digital sensor such as a one or morecharge-coupled device (CCD) or a complementary metal oxide semiconductor(CMOS) device store image data. Endoscopic imaging systems may providetwo- or three-dimensional images to the viewer. Two-dimensional imagesmay provide limited depth perception. Three-dimensional stereoendoscopic images may provide the viewer with more accurate depthperception. Stereo endoscopic instruments employ stereo cameras tocapture stereo images of the patient anatomy. An endoscopic instrumentmay be a fully sterilizable assembly with the endoscope cable, handleand shaft all rigidly coupled and hermetically sealed.

FIG. 1C is a perspective view of the surgeon's console 16. The surgeon'sconsole 16 includes a left eye display 32 and a right eye display 34 forpresenting the surgeon S with a coordinated stereo view of the surgicalenvironment that enables depth perception. The displayed image of thesurgical environment may be obtained from an imaging system such as theendoscopic imaging system. Additionally or alternatively, the displayedimage of the surgical environment may include images from anatomicmodels created from pre-operative or intra-operative image data sets.Pre-operative or intraoperative image data set of the patient anatomymay be obtained using external or non-invasive imaging technology suchas computerized tomography (CT), magnetic resonance imaging (MRI),fluoroscopy, thermography, ultrasound, optical coherence tomography(OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-rayimaging, or the like. Software alone or in combination with manual inputis used to convert the recorded images into segmented two dimensional orthree dimensional composite models representing a partial or an entireanatomical organ or anatomical region. An image data set is associatedwith the composite representation. The images used to generate thecomposite representation may be recorded preoperatively orintra-operatively during a clinical procedure. The pre-operative orintra-operative image data may be presented as two-dimensional,three-dimensional, or four-dimensional (including e.g., time based orvelocity based information) images or as images from models created fromthe pre-operative or intra-operative image data sets. Images fromdifferent imaging modalities may be displayed one at a time (e.g., thesurgeon may toggle through the different modality images), may bedisplayed in parallel (e.g., in multiple windows of a composite display)or one may be overlaid or superimposed on the other.

The console 16 further includes one or more input control devices 36,which in turn cause the teleoperational assembly 12 to manipulate one ormore instruments or the endoscopic imaging system. The input controldevices 36 can provide the same degrees of freedom as their associatedinstruments 14 to provide the surgeon S with telepresence, or theperception that the input control devices 36 are integral with theinstruments 14 so that the surgeon has a strong sense of directlycontrolling the instruments 14. To this end, position, force, andtactile feedback sensors (not shown) may be employed to transmitposition, force, and tactile sensations from the instruments 14 back tothe surgeon's hands through the input control devices 36. Input controldevices 37 are foot pedals that receive input from a user's foot.Optionally, input control device 38 may include a touch-based inputdevice such as a computer tablet. Optionally, a gesture-based interfacemay be included in the console 16.

During a teleoperational procedure, entirely virtual and/or augmentedreality images may be provided to the surgeon S and/or other surgicalpersonnel to provide a more extensive view of the surgical environment,provide additional information about the patient or the procedure,and/or provide additional controls for use during the procedure. Varioussystems and methods for providing virtual or augmented reality imagesduring a teleoperational procedure are disclosed in U.S. Pat. No.8,520,027, filed May 14, 2010, disclosing “Method and System ofSee-Through Console Overlay;” International Publication Number WO2014/176403, filed Apr. 24, 2014, disclosing “Surgical Equipment ControlInput Visualization Field;” U.S. Pat. No. 8,398,541, filed Aug. 11,2008, disclosing “Interactive User Interfaces for Robotic MinimallyInvasive Surgical Systems;” U.S. Pat. No. 9,788,909, filed Nov. 11,2013, disclosing “Synthetic Representation of a Surgical Instrument;”and U.S. Pat. No. 9,858,475, filed May 14, 2010 disclosing “Method andSystem of Hand Segmentation and Overlay Using Depth Data,” which areincorporated by reference herein in their entirety.

FIG. 2 is a display 100 illustrating surgical environment image 102,which in this example is an interior patient anatomy image obtained byan endoscopic imaging system (e.g., system 15, 28) augmented with acursor image 104 of a real-time image of a user's extremity, which inthis example is a right hand of the surgeon. The image of the surgeon'shand may be obtained by a camera 27 a-c or by the GBI system 29 and usedas a cursor to indicate the current position for interaction with anobject in the image 102. The endoscopic image 102 includes an image ofan instrument 106 in the surgical environment. In this example theinstrument 106 is a retractor. When the teleoperational system is in anadjustment mode, the surgeon S may move his right hand, thereby movingthe cursor 104 to interact with the image of the instrument 106 (e.g., aretractor), to cause an adjustment in the actual position of theretractor in the surgical environment. The cursor image 104 of thesurgeon's hand allows the surgeon to visualize his hand virtuallyselecting and moving the retractor 106. The movement of the retractor inthe surgical environment may be generated by the teleoperational systemin response to the commanded motion of the surgeon's hand. In variousalternative embodiments, a cursor image may be a static (i.e.,previously captured) image of the user's extremity (e.g., a hand,finger, foot) or another type of static cursor symbol depicting aportion of a human anatomy (e.g., an image of an eye or a head).

FIG. 3 illustrates a method 150 of adjusting a virtual control elementwith a cursor image of a user's extremity. The method 150 is illustratedin FIG. 3 as a set of operations or processes. Not all of theillustrated processes may be performed in all embodiments of method 150.Additionally, one or more processes that are not expressly illustratedin FIG. 3 may be included before, after, in between, or as part of theillustrated processes. In some embodiments, one or more of the processesof method 150 may be implemented, at least in part, in the form ofexecutable code stored on non-transitory, tangible, machine-readablemedia that when run by one or more processors (e.g., the processors ofcontrol system 20) may cause the one or more processors to perform oneor more of the processes.

At a process 152, a virtual control element is displayed. The virtualcontrol element may be a sliding adjuster, a toggle switch, a dial, orother element for controlling a binary system function (e.g. on/off) ora variable system function (e.g., power level, brightness level, soundlevel, frequency level) of a component of the teleoperational system oran auxiliary equipment component in the surgical environment. At aprocess 154, a cursor image of a user (e.g. the surgeon S) body part(e.g., the hand of surgeon S) used for controlling the virtual controlelement is displayed. At a process 156, a gesture based interface (e.g.GBI 29) receives input from the surgeon S by registering movement of theuser's hand virtually manipulating the virtual control element as thereal-time cursor image of the user's hand interacts with the virtualcontrol element. At a process 158, the binary system function orvariable system function is changed or adjusted based on the movement ofthe user's hand. The method 150 is further illustrated with reference toFIGS. 4 and 5.

With reference to FIG. 4A, an image 200 includes a cursor image of auser's extremity (e.g., a right hand) 202 interacts with a virtualcontrol element 204, which in this example includes a virtual slider206. A graphical element such as a graphical identifier 208 of thesystem controlled by the control element 204 may also be included. Inthis example, the identifier 208 is a textual description, but in otherexamples the identifier 208 may be a pictorial representation of thesystem controlled by the control element. In this example, the virtualcontrol element 204 may control the variable level of power, sound,frequency, or other characteristic of an auxiliary system. The auxiliarysystem may be, for example, a power generator, a speaker, a displayscreen, or an irrigation system. As the user's hand moves the slider 206to the right, the power level of the generator increases and as theslider moves to the left, the power level decreases. The user's hand maybe tracked by a GBI or a touch-based input system. Additionally, thereal-time cursor image of the user's hand 202 may be generated by acamera 27 a-c to provide the user with spatial awareness of his handrelative to the virtual control element 204. Optionally, the images202-208 may be superimposed on, integrated with, or otherwise augment animage of either the internal patient anatomy or an image of the surgicalenvironment external of the patient anatomy.

FIG. 4B illustrates an image 220 of the real-time cursor image 202interacting with a real-time image of an auxiliary component 222 in thesurgical environment. In this example, the image 220 captures thesurgical environment which includes the auxiliary component 222. Theimage of the auxiliary component in the image 220 serves as a virtualcontrol element for the actual auxiliary component 222 in the surgicalenvironment. For example, if the auxiliary component 222 is a highfrequency power generator, the surgeon may gesture with his hand in apredetermined motion toward the auxiliary component 222 in the surgicalenvironment to change the power level of the generator. An upward handor finger gesture may correspond to an increase in power level and adownward hand or finger gesture may correspond to a decrease in powerlevel. Alternatively, the auxiliary component 222 may include a powercontrol knob 224, and a clockwise hand gesture toward the knob 224 maycorrespond to an increase in power level with a counter-clockwise handgesture may correspond to a decrease in power level. In alternativeembodiments, the auxiliary system may be, for example, a speaker, adisplay screen, or an irrigation system. The user's hand may be trackedby a GBI or a touch-based input system. Additionally, the real-timecursor image of the user's hand 202 may be generated by a camera 27 a-cto provide the user with spatial awareness of his hand relative to thevirtual control element.

FIG. 5 illustrates an image 230 of a surgical environment with areal-time cursor image 232 interacting with an information icon 234 adisplayed over the surgical environment. In this example, the image 230captures the surgical environment which includes instruments 236 a, 236b, and 236 c. Information icon 234 a is displayed near icon 234 a.Information icon 234 b is displayed near icon 234 b. Information icon234 c is displayed near icon 234 c. As the cursor image 232 contacts orcomes into proximity with one of the information icons 234 a-b,information about the respective instrument 236 a-c is displayed in aninformation cloud. For example, information cloud 238 is displayed whenthe cursor image 232 interacts with the information icon 234 a toprovide information about the tool 236 a. The provided information mayinclude, for example, the type of instrument, the activation status ofthe instrument, instructions about operating or trouble-shooting theinstrument, and/or buttons activatable by the cursor 232 to effect theoperation of the instrument 236 a.

In an alternative embodiments, the cursor image of the user's extremitymay be a real-time image of the user's foot as it moves between pedalinputs 37. The real-time image of the foot and pedals may be obtained bya camera 27 a-c and may be presented as a separately displayed image, asa picture-in-picture within the current endoscopic image, or behind asemi-transparent current endoscopic image to provide the surgeon withthe sense that he is looking through the console 16 to view his foot asit moves toward a different pedal. In still another alternativeembodiment, the image of the user's extremity may be a real-time imageof the user's hand as it transitions out of or into engagement with theinput control devices 36. The real-time image of the hand and controldevices 36 may be obtained by a camera 27 a-c and may be presented as aseparately displayed image, as a picture-in-picture within the currentendoscopic image, or behind a semi-transparent current endoscopic imageto provide the surgeon with the sense that he is looking through theconsole 16 to view his hand as it moves into or out of the controldevice 36. Allowing the surgeon to see his hands or feet as theytransition between positions may boost the surgeon's confidence that hishands and feet, which are not directly visible due to the console 16blocking the user's view, are making the correct engagement with theinput devices 36, 37.

FIG. 6 illustrates a method 250 of patient marking using a gesture basedinput device. The method 250 is illustrated in FIG. 6 as a set ofoperations or processes. Not all of the illustrated processes may beperformed in all embodiments of method 250. Additionally, one or moreprocesses that are not expressly illustrated in FIG. 6 may be includedbefore, after, in between, or as part of the illustrated processes. Insome embodiments, one or more of the processes of method 250 may beimplemented, at least in part, in the form of executable code stored onnon-transitory, tangible, machine-readable media that when run by one ormore processors (e.g., the processors of control system 20) may causethe one or more processors to perform one or more of the processes.

At a process 252, a surgical environment including a virtual markingelement is displayed. The virtual marking element may be used, forexample, to indicate the surgeon's preferred entry port locations or maymark anatomic features of the patient. At a process 254, a cursor imageof a user (e.g. the surgeon S) body part (e.g., the hand of surgeon S)used for interacting with the virtual marking element is displayed. At aprocess 256, a gesture-based interface (e.g. GBI 29) receives input fromthe surgeon S by registering movement of the user's hand virtuallyinteracting with the marking element as the real-time cursor image ofthe user's hand interacts with the virtual marking element. For example,the movement of the user's hand may be used to create a new marker ormove a marker from a default location. At a process 258, the patientanatomy is marked (e.g. with light, ink, or other marking material)based on the position of the marking element. The method 250 is furtherillustrated with reference to FIG. 7.

With reference to FIG. 7, a surgical environment image 300 includes anexterior image 301 of a patient anatomy. A cursor image of a user'sextremity (e.g., a right hand) 202 interacts with virtual markingelements 302, 304, 306. The virtual marking elements may be created by agesture of the user's hand or may be created in a default location. Theuser's hand may interact with a virtual marking element 306 to move themarking element to a different location than where it was originallycreated. In this example the virtual marking elements 302, 304, 306 maybe used to mark port locations on the image 301 of the patient anatomy.The virtual marking elements be virtually dragged relative to the image301 of the patient anatomy by the tracked motion of the user's hand. Theuser's hand may be tracked by a GBI or a touch-based input system.Additionally, the real-time cursor image of the user's hand 202 may begenerated by a camera 27 a-c to provide the user with spatial awarenessof his hand relative to the virtual marking elements 302, 304, 306.After the virtual marking elements 302, 304, 306 are establishedrelative to the image 301 of the patient anatomy, corresponding actualmarks may be made with light, ink, or another marking medium on theanatomy of the patient to mark the locations of entry ports prior to asurgical procedure. The visual marking elements may, for example,indicate candidate incision locations which may be evaluated by thecontrol system to provide feedback on the feasibility of the incisionlocations by evaluating reachable workspace and the likelihood ofinternal or external collisions. The virtual marking elements and thesurrounding areas may be color coded based on the predicted feasibilitymeasures.

FIG. 8 illustrates a method 450 of displaying an image of an internalpatient anatomy at least partially surrounded by the displayed image ofthe external patient anatomy. The method 450 is illustrated in FIG. 8 asa set of operations or processes. Not all of the illustrated processesmay be performed in all embodiments of method 450. Additionally, one ormore processes that are not expressly illustrated in FIG. 8 may beincluded before, after, in between, or as part of the illustratedprocesses. In some embodiments, one or more of the processes of method450 may be implemented, at least in part, in the form of executable codestored on non-transitory, tangible, machine-readable media that when runby one or more processors (e.g., the processors of control system 20)may cause the one or more processors to perform one or more of theprocesses.

At a process 452, a first image from a first imaging device is received.The first image may be an internal view of the patient anatomy,received, for example, from an endoscopic device. At a process 454, asecond image from a second imaging device is received. The second imagemay be an external view of the patient anatomy and/or the surgicalenvironment surrounding the patient anatomy. At a process 456, the firstimage is displayed in spatial context relative to the second image. Forexample, as shown in FIG. 9, a display 500 includes an image 502 of theinternal patient anatomy at least partially surrounded by the displayedimage 504 of the external patient anatomy. The image 502 may be obtainedby an endoscopic imaging system (e.g., system 15, 28), and the image 504may be obtained by an imaging system such a camera 27 a-c. The viewingorientation of the internal and external views may be aligned. Forexample, the external view may be digitally rotated to share the sameroll angle as the internal (e.g., endoscopic) view. Additionally oralternatively, the pitch and/or yaw of the external view may be alignedwith the viewing direction of the internal view so that internal andexternal motions can be intuitively controlled from the same referenceframe.

One or more elements in embodiments of the invention may be implementedin software to execute on a processor of a computer system such ascontrol processing system. When implemented in software, the elements ofthe embodiments of the invention are essentially the code segments toperform the necessary tasks. The program or code segments can be storedin a processor readable storage medium or device that may have beendownloaded by way of a computer data signal embodied in a carrier waveover a transmission medium or a communication link. The processorreadable storage device may include any medium that can storeinformation including an optical medium, semiconductor medium, andmagnetic medium. Processor readable storage device examples include anelectronic circuit; a semiconductor device, a semiconductor memorydevice, a read only memory (ROM), a flash memory, an erasableprogrammable read only memory (EPROM); a floppy diskette, a CD-ROM, anoptical disk, a hard disk, or other storage device, The code segmentsmay be downloaded via computer networks such as the Internet, Intranet,etc.

Note that the processes and displays presented may not inherently berelated to any particular computer or other apparatus. Variousgeneral-purpose systems may be used with programs in accordance with theteachings herein, or it may prove convenient to construct a morespecialized apparatus to perform the operations described. The requiredstructure for a variety of these systems will appear as elements in theclaims. In addition, the embodiments of the invention are not describedwith reference to any particular programming language. It will beappreciated that a variety of programming languages may be used toimplement the teachings of the invention as described herein.

While certain exemplary embodiments of the invention have been describedand shown in the accompanying drawings, it is to be understood that suchembodiments are merely illustrative of and not restrictive on the broadinvention, and that the embodiments of the invention not be limited tothe specific constructions and arrangements shown and described, sincevarious other modifications may occur to those ordinarily skilled in theart.

1-27. (canceled)
 28. A system comprising: a processor; and a memoryhaving computer readable instructions stored thereon, the computerreadable instructions, when executed by the processor, cause the systemto: display a surgical environment image, wherein the surgicalenvironment image includes a virtual control element for controlling acomponent of a surgical system, and wherein the virtual control elementincludes a real-time image of the component of the surgical system inthe surgical environment image; display an image of a body part of auser, the body part used to interact with the virtual control element;receive a gesture of the body part of the user in a predeterminedmotion, via a gesture based input device registering movement of thebody part of the user, while the body part interacts with the virtualcontrol element; and adjust a setting of the component of the surgicalsystem based on the received gesture.
 29. The system of claim 28 whereinthe virtual control element includes a graphical element superimposed onthe surgical environment image.
 30. The system of claim 28 furthercomprising the gesture based input device, wherein the gesture basedinput device is configured to receive a three dimensional user input.31. The system of claim 28 further comprising the gesture based inputdevice, wherein the gesture based input device includes at least one ofa tablet device or a user wearable device.
 32. The system of claim 28wherein the body part used to provide input to the gesture based inputdevice is at least one of a user hand or a user foot.
 33. The system ofclaim 28 wherein the surgical environment image is displayed on ahead-mounted display device.
 34. A system comprising: a processor; and amemory having computer readable instructions stored thereon, thecomputer readable instructions, when executed by the processor, causethe system to: display a surgical environment image, wherein thesurgical environment image includes a virtual marking element; display abody part of a user, the body part used to interact with the virtualmarking element; receive a user input, via a gesture based input deviceregistering movement of the body part of the user, while the body partinteracts with the virtual marking element; generate a patient anatomymark on a patient anatomy based on the received user input, wherein thevirtual marking element indicates a potential incision location on thepatient anatomy; and evaluate a feasibility of the potential incisionlocation.
 35. The system of claim 34 wherein the virtual marking elementincludes a plurality of virtual port markers for marking locations of aplurality of anatomic entry ports on the patient anatomy.
 36. The systemof claim 34 wherein the surgical environment image is an endoscopicimage inside the patient anatomy.
 37. The system of claim 34 wherein thesurgical environment image is an external image of the patient anatomy.38. The system of claim 34 further comprising the gesture based inputdevice, wherein the gesture based input device is configured to receivea three dimensional user input.
 39. The system of claim 34 furthercomprising the gesture based input device, wherein the gesture basedinput device includes at least one of a tablet device or a user wearabledevice.
 40. The system of claim 34 wherein the body part used to provideinput to the gesture based input device is at least one of a user handor a user foot.
 41. A system comprising: a processor; and a memoryhaving computer readable instructions stored thereon, the computerreadable instructions, when executed by the processor, cause the systemto: display a first image on a display while a patient is located in asurgical environment, wherein the first image includes: an image of aninternal patient anatomy received from a first imaging device, a virtualcontrol element for controlling a component of a surgical system in thesurgical environment, and a real-time image of the component of thesurgical system; display a second image on the display while the patientis located in the surgical environment, wherein the second imageincludes an image of the surgical environment external of the patientanatomy received from a second imaging device, wherein the displayedfirst image is at least partially surrounded by the displayed secondimage; receive a gesture of a body part of a user in a predeterminedmotion, via a gesture based input device registering movement of thebody part of the user, while the body part interacts with the virtualcontrol element; and adjust a setting of the component of the surgicalsystem based on the received gesture.
 42. The system of claim 41 whereinthe second imaging device is positioned on an instrument in the surgicalenvironment.
 43. The system of claim 41 wherein the display is includedin at least one of a head-mounted device or a patient-side device. 44.The system of claim 41 wherein the first imaging device is an endoscopicdevice having a view axis and the second imaging device is orientedalong the view axis.
 45. The system of claim 41 wherein the image of theinternal patient anatomy is preoperatively obtained by the first imagingdevice.
 46. The system of claim 41 wherein the image of the internalpatient anatomy is at least one of a CT image or an X-ray image.
 47. Thesystem of claim 41, wherein receiving the gesture of the body part ofthe user includes receiving a gesture from at least one of a user handor a user finger.